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EXECUTIVE SUMMARY 



INTRODUCTION 

This report summarizes the Year 2 evaluation study results of the Students and Teachers Accessing 
Real-time Knowledge (STARK) Program in the West Orange Consolidated Independent School 
District (CSID). The overall purpose of the evaluation was twofold: (a) to provide formative evaluation 
data to the participant schools to serve as a basis for improvement planning and as documentation of 
their accomplishments to demonstrate progress; and (b) to provide cumulative evidence of the 
implementation progress and outcomes of the participant schools as well as identification of 
exemplary programs. 

The context for the evaluation consists of 10 schools receiving “Foundation” training developed by the 
Brazos-Sabine Connection and integration training provided by West Orange Collaborative. The 
Integration training places emphasis on employing technology in the classroom to support student- 
centered teaching methods that promote higher-level learning outcomes. It builds on Foundation 
training that deals directly with developing skills in using technology in the classroom. 

The present 2002-2003 evaluation data were collected from 1 0 schools over the academic year. The 
data were compared to 2001-2002 baseline conditions to gauge program implementation progress. 

RESEARCH QUESTIONS 

The STARK Program evaluation was structured around six primary research questions that focused 
on classroom practices, degree and type of technology use, academically focused time, student 
engagement, student achievement, and school climate. Also of interest were teacher ability with, use 
of, and attitudes toward technology. 

1 . To what degree and how is technology integrated with classroom instruction by teachers in 
general and those who have received Foundation and/or integration training through the 
Enhanced Learning Academy? 

2. To what degree do integration-trained teachers use methodologies that, consistent with STARK 
Program goals, stress higher-order learning and student-centered learning activities? 

3. To what degree have teachers acquired the technology skills specified in the Texas state 
standards? 

4. What are school outcomes in school climate and teacher uses of and attitudes toward 
technology? To what extent do these variables (a) reflect Foundation training and STARK 
Program goals, and (b) correlate with one another and with implementation success? 

5. How are program activities being directed to increase student learning and specifically, to raising 
achievement on the Texas Essential Knowledge and Skills (TEKS)? (after Year 2) 

6. What factors appear most instrumental in determining schools’ success at achieving the goals 
and overall implementation of the STARK Program? 
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DESIGN 



The evaluation design was based on both quantitative and qualitative data collected from classroom 
observations and teacher surveys. Participating schools were the 1 0 schools receiving “Foundation” 
training developed by the Brazos-Sabine Connection and integration training provided by West 
Orange Collaborative. 



INSTRUMENTATION 

Six instruments were used to collect the evaluation data: three classroom observation measures 
and three teacher surveys. The instruments used in the study are fully developed and validated. 
They are as follows: 



Classroom Observation Measures 

Observations were made focusing on targeted classes (scheduled visits) using three instruments. 



• School Observation Measure (SOM): Examines frequency of usage of 24 instructional strategies. 

• Survey of Computer Use (SCU): Examines availability and student use of technology and software 
applications. 

• Rubric for Student-Centered Activities (RSCA) : Rates the degree of learner engagement in cooperative 
learning , project-based learning , higher-level questioning, experiential/hands-on learning, student 
independent inquiry/research, student discussion, and students as producers of knowledge using 
technology. 

Surveys 



• T eacher T echnology Questionnaire (TTQ) ; Collects teacher perceptions of computers and technology. 

• School Climate Inventory (SCI): Assesses school staff perceptions of school climate on seven 
dimensions: Order, Leadership, Environment, Involvement, Instruction, Expectations, and Collaboration. 

• Technology Skills Assessment (TSA): Assesses the perceived technological abilities of the teachers in 
these areas: Computer Basics, Software Basics, Multimedia Basics, Internet Basics, Advanced Skills, 
Using Technology for Learning, and Policy and Ethics. 



PROCEDURE 

Data for this evaluation study were collected in spring 2003. The SOM, SCU and RSCA were 
completed for the targeted observations. These consisted of prearranged one-hour sessions in which 
teachers who participated in the program’s technology training demonstrated a prepared lesson using 
technology-notes forms were completed every 1 5 minutes of the lesson. A total of 52 classroom visits 
were completed. The teacher surveys (TTQ, SCI, and TSA) were administered in May 2003 during 
faculty meetings at each school. 



RESULTS 

Below is a brief summary of the results grouped by Classroom Observation Results and Survey 
Results. 
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Classroom Observation Results 



The data for 52 classroom observations were collected with SOMs, SCUs, and RSCAs during 
prearranged sessions in which teachers implemented a lesson using technology. Results from each 
measure are described in the section below. 



School Observation Measure (SOM®) 

The SOM results revealed significant changes between baseline and Year 2 observations. 

These changes included using a greater variety of instructional strategies as well as using more 
student-centered practices. The largest differences were seen with regard to classroom use of 
technology. During Year 1 , computer use was seen in only 30% or less of the observations, whereas 
Year 2 data revealed computer use in over 65% of the visits. The data suggest that the STARK 
program is having a positive impact on increasing teacher’s use of student-centered learning 
techniques, integration of technology into instruction, and higher levels of academically focused class 
time and student attention. 

Rubric for Student-Centered Activities (RSCA) 

The results show that the RSCA strategies were observed in approximately one-third to one-half 
of the classroom observations - a slight increase (1 6%) from the baseline results. Also promising is 
the positive trend in the quality of strategy implementation - with all mean scores being above 
average. Although not significant, the mean scores for five of the seven RSCA strategies were higher 
- indicating continued growth toward teacher competency with non-traditional instructional strategies. 
The most striking RSCA result is the increased usage of technology to support student-centered 
activities. 

Survey of Computer Use (SCU®) 

The SCU results suggest that progress was made between Year 1 and Year 2 with regard to 
technology integration efforts. Specifically, students had more access to computers and digital tools; 
computer skills; and were more frequently using a variety of software tools. There were significant 
increases in student use of presentation software and use of the Internet, however, these and other 
software applications were still only seen in about 25% of the visits. The most striking result was the 
significant increase - indicating that teachers have increased their understanding and use of 
technology as a tool to enhance student learning. 



Survey Results 

Three surveys (SCI, TTQ, and TSA) were administered to the teachers during a faculty meeting 
held at each school in late May 2003. Results for the three instruments were examined for Year 1 vs. 
Year 2 differences. Data from the TTQ and TSA were also compared based on the amount of 
technology training received: no or partial training vs. full training. Results of the three surveys are 
presented below. 

School Climate Inventory (SCI) 

The moderate gain in SCI results suggests a positive trend in teacher attitudes about their school 
climate. This was particularly true for Involvement and Instruction, which were both significantly higher 
than the previous year. A slight decrease was seen for Order and Environment; however, the 
resulting means were still higher than the national norms. The data also suggest that schools may 
want to explore ways to increase student and parent-level involvement with educational issues. 

Overall SCI results once again reflect school environments that are generally open and supportive of 
reform initiatives focused toward improved educational opportunities. 
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Teacher Technology Questionnaire (TTQ) 

The TTQ results for 2002-2003 show significant positive changes from the previous year in 
teacher perceptions of their readiness to integrate technology. This readiness is reflected in the high 
level of reported confidence with regard to teacher computer skills being adequate enough to conduct 
classes that have students use technology. There was also strong agreement that the teachers could 
readily obtain answers to technical problems and that school computers were kept in good working 
condition. However, approximately 50% of teachers were still less positive with regard to the impact 
of technology on their instruction. When examining the TTQ results by amount of technology training, 
two significant differences were revealed - both suggesting that the training had an educationally 
sound impact on teacher readiness to integrate technology and teacher belief that technology has a 
positive impact on classroom instruction. 

Technology Skills Assessment 

The TSA results revealed an overall increase in teacher confidence to complete a wide variety of 
computer-related tasks. The greatest level of confidence was with computer, software, Internet and 
multimedia basics. The teachers who had completed the technology training were significantly more 
confident with their ability to use technology as a learning tool. 



CONCLUSIONS 



The conclusions of the present study will be presented in association with each of the major 
research questions in the respective sections below. 



To what degree and how is technology integrated with classroom instruction by teachers in 
general* and those who have received Foundation and/or integration training through the 
Enhanced Learning Academy? 

Classroom observation data from both the SOM and SCU revealed significant differences in the 
degree and the way in which teachers integrate technology into their teaching. The most striking was 
a 40% increase in the use of computers as an instructional delivery tool and the 50% increase in 
student use of technology as a learning tool, which was seen in over 65% of the classroom 
observations. Another noteworthy increase in the use of computers is seen in data from the RSCA, 
which revealed that teachers integrated technology with approximately one third or more of their 
student-centered activities, as compared to less than 1 0% the previous year. Although students had 
more access to computers and digital tools, significant increases in the types of computer activities 
observed were only found for the use of Internet and presentation software. Word processing and 
educational software were the other two most frequently observed applications. Despite the limited 
scope of technology tools that were used to support classroom instruction, a dramatic improvement 
was seen in teacher ability to create and implement activities that meaningfully integrated the use of 
technology to enhance student learning. Specifically, meaningful (from somewhat to very meaningful) 
use of computers was only seen in 9.3% of the Year 1 observations as compared to 41 .2% in Year 2. 
These results are highly suggestive that the STARK Program is effectively impacting technology 
integration efforts in the target schools. 

'Note. The Year 2 classroom observations were once again conducted during pre-arranged visits to classrooms of 
teachers who had received technology training, therefore, the results reflect trained teachers rather than teachers in general. 

The Year 3 data will include all teachers. 

To what degree do integration-trained teachers use methodologies that, consistent with 
STARK Program goals, stress higher-order learning and student-centered learning activities? 

In over two-thirds of the classroom observations, teachers assumed the role of coaches or 
facilitators and technology was being used as a tool to support learning. Other activities that foster 
critical thinking and student engagement were seen in over 50% of the visits. These included 
experiential hands-on learning (57.7%), teacher provision of higher-level instructional feedback to 
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students (55.8%), and teacher use of higher-level questioning strategies (53.8%). Although no 
significant increases were found when specifically examining the quality with which teachers 
implement student-centered activities, it is promising to note that all mean scores were above average 
- with five being higher than Year 1 ratings. The data indicate continued growth toward teacher 
competency with non-traditional instructional strategies. However, professional development efforts 
may want to increase the focus on better preparing teachers to implement these methods to increase 
the frequency with which they are utilized. 

To what degree have teachers acquired the technology skills specified in the Texas State 
standards? 

Impressive increases were seen in teacher confidence in all seven categories of technology 
skills. Nearly all (greater than 90%) of the teachers indicated that they could easily complete basic 
computer and software tasks and that they understood the policy and ethics of technology use. 
Seventy-five percent felt confident when using the Internet and over 60% reported that they could 
easily or very easily complete multimedia and/or advanced tasks, and use technology for learning. As 
would be expected, the teachers who had completed the technology training were significantly more 
confident with their ability to use technology as a learning tool - again an indication of successful 
teacher training through the STARK Program. 

What are school outcomes in school climate and teacher uses of and attitudes toward 
technology? To what extent do these variables (a) reflect Foundation training and STARK 
Program goals, and (b) correlate with one another and with implementation success? 

Results from the School Climate Inventory were again above national norms, suggesting that the 
environment of the 1 0 participating schools was supportive of school-wide initiatives such as 
technology integration. This positive climate was evidenced in the significant increases in frequency 
of classroom use of technology and teacher ability to create and implement meaningful computer 
activities. Positive teacher attitudes are also reflected in the TTQ results, which showed significantly 
higher teacher readiness to integrate technology and more confidence to conduct classes that have 
students use technology. A contributing factor to this increased confidence could perhaps be the 
increased level of computer literacy as reported on the TSA, plus the ready access to technical 
support provided by the schools and district. Significant differences were again revealed when 
examining results by the degree of training received - suggesting that the STARK Program had an 
educationally sound impact on teacher readiness to integrate technology and teacher belief that 
technology has a positive impact on classroom instruction. The overall results do suggest a positive 
trend in achieving the STARK Program goals and successful implementation, however the current 
efforts need to continue or increase to impact the approximately 50% of teachers who were still less 
positive with regard to the impact of technology on their instruction. 

What factors appear most instrumental in determining schools’ success at achieving the 
goals and overall implementation of the STARK Program? 

Below is a list of key factors from the Year 2 study that appeared to have influenced the progress 
being made toward achievement of the STARK Program implementation and goals. As seen, the 
factors cross key elements that are critical for program success: 

- Significant differences in instructional practices 

- Significant differences in the quality of instructional practices 
Significant differences in school climate 

- Significant differences in teacher attitudes and beliefs regarding technology integration 
Significant differences in teacher computer skills 

Overall, the triangulation of positive changes suggests a strong synergy between the diverse 
factors that resulted in impressive progress toward achieving the STARK Program goals during the 
second year of implementation. As seen, the schools had positive climates, teacher technological 
competence increased, and classroom practices engaged students in meaningful technology- 
supported activities. These positive results suggest that program implementation for Year 3 should 
emulate Year 2, but with increased intensity to ensure all teachers are impacted. 
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WEST ORANGE COLLABORATIVE 



STARK PROGRAM 

EVALUATION REPORT 



This report summarizes the 2002-2003 evaluation study results of the Students and Teachers 
Accessing Real-time Knowledge (STARK) Program in West Orange Consolidated Independent 
School District (CSID). The overall purpose of the evaluation was twofold: (a) to provide formative 
evaluation data to the participant schools to serve as a basis for improvement planning and as 
documentation of their accomplishments to demonstrate progress; and (b) to provide cumulative 
evidence of the implementation progress and outcomes of the participant schools as well as 
identification of exemplary programs. 

The context for the evaluation consists of 1 0 schools receiving two levels of educational 
technology training. For the first level, teachers complete the online “Foundation” training modules 
developed by the Brazos-Sabine Connection. The second level involves teachers completing 
integration training provided during Saturday training sessions and an online University of Texas at 
Brownsville course developed by Karon Tarver of West Orange Collaborative. The Integration training 
places emphasis on employing technology in the classroom to support student-centered teaching 
methods that promote higher-level learning outcomes. It builds on Foundation training that deals 
directly with developing skills in using technology in the classroom. 

The evaluation concentrated on the progress of the 1 0 schools in achieving STARK Program 
goals and raising student achievement. It is expected that the program will be implemented and 
monitored for multiple years. The present Year 2 evaluation examined progress over the 2002-2003 
academic year. These data are compared to 2001 -2002 baseline conditions to gauge program 
implementation progress. Suggestions for improvement are offered on the basis of the findings. 
Specific evaluation questions that guided the methodology, data collection, and reporting are listed in 
the next section. 
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RESEARCH QUESTIONS 



The STARK Program evaluation was structured around six primary research questions that 
focused on classroom practices, degree and type of technology use, academically focused time, 
student engagement, student achievement and school climate. Also of interest were teacher ability 
with, use of, and attitudes toward technology. The detailed questions are listed below: 

• To what degree and how is technology integrated with classroom instruction by teachers in 
general and those who have received Foundation and/or integration training through the 
Enhanced Learning Academy? 

• To what degree do integration-trained teachers use methodologies that, consistent with 
STARK Program goals, stress higher-order learning and student-centered learning activities? 

• To what degree have teachers acquired the technology skills specified in the Texas state 
standards? 

• What are school outcomes in school climate and teacher uses of and attitudes toward 
technology? To what extent do these variables (a) reflect Foundation training and STARK 
Program goals, and (b) correlate with one another and with implementation success? 

• How are program activities being directed to increase student learning and specifically, to 
raise achievement on the Texas Essential Knowledge and Skills (TEKS)? (after Year 2) 

• What factors appear most instrumental in determining schools’ success at achieving the 
goals and overall implementation of the STARK Program? 

EVALUATION DESIGN AND MEASURES 

The evaluation was conducted during the spring semester of the 2002-2003 academic year. The 
basic evaluation design for Year 2 reflected that of the Year 1 baseline study to enable two types of 
comparative analyses to be conducted: 1) Year 1 (baseline) vs. Year 2 and 2) no/partial vs. full 
technology training completed by teachers. Both quantitative and qualitative data were collected at 
each of the 1 0 schools by trained observers (e.g., retired teachers, district staff). The observers spent 
the major part of their time visiting classrooms (using three instruments to be described below), but 
also administered surveys to teachers, and monitored Benchmark development and use in assessing 
implementation progress. A description of the instruments and data collection procedures is 
presented below. 
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Instrumentation 



Six instruments were used to collect the evaluation data: three classroom observation measures 
and three teacher surveys. The instruments used in the study are fully developed and validated. 

They are as follows: 

Classroom Observation Measures 

Trained observers conducted targeted classroom visits to collect frequency data regarding 
observed instructional practices. The visits were considered targeted because observations were 
scheduled in advance with randomly selected teachers (from those who participated in the program’s 
technology training) who were instructed to deliver a lesson that integrates the use of technology. The 
data collection instruments were the School Observation Measure (SOM), the Survey of Computer 
Use (SCU), and the Rubric for Student-Centered Activities (RSCA ). The SOM was used to collect 
data regarding overall classroom activities, the SCU for student use of computers, and the RSCA to 
capture more detailed information about student-centered activities during the targeted observations. 
The classroom observation instruments are described below. 

SOM. The SOM was developed to determine the extent to which different common and 
alternative teaching practices are used throughout an entire school (Ross, Smith, & Alberg, 1999). 

The standard, or whole-school SO!\/P procedure involves observers’ visiting 1 0-1 2 randomly selected 
classrooms, for 1 5 minutes each, during a three-hour visitation period. The observer examines 
classroom events and activities descriptively, not judgmentally. Notes are taken relative to the use or 
nonuse of 24 target strategies. At the conclusion of the three-hour visit, the observer summarizes the 
frequency with which each of the strategies was observed across all classes in general on a data 
summary form. The frequency is recorded via a 5-point rubric that ranges from (0) Not Observed to 
(4) Extensively. Two global items use three-point scales (low, moderate, high) to rate, respectively, 
the use of academically-focused instructional time and degree of student attention and interest. 
Targeted observations were conducted to examine classroom instruction during prearranged one- 
hour sessions in which randomly selected teachers demonstrated a prepared lesson using 
technology. The notes forms were completed every 1 5 minutes of the lesson then were summarized 
on a SOM Data Summary Form. 

Expanded Rubric (ER) in Yr.1 
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To ensure the reliability of data, observers receive a manual providing definitions of terms, 
examples and explanations of the target strategies, and a description of procedures for completing the 
instrument. The target strategies include traditional practices (e.g., direct instruction and independent 
seatwork) and alternative, predominately student-centered methods associated with educational 
reforms (e.g., cooperative learning, project-based learning, inquiry, discussion, using technology as a 
learning tool). The strategies were identified through surveys and discussions involving policy 
makers, researchers, administrators, and teachers, as those most useful in providing indicators of 
schools’ instructional philosophies and implementations of commonly used reform designs (Ross, 
Smith, Alberg, & Lowther, 2001). 

After receiving the manual and instruction in a group session, each observer participates in 
sufficient practice exercises to ensure that his/her data are comparable with those of experienced 
observers. In a reliability study (Lewis, Ross, & Alberg, 1999), pairs of trained observers selected the 
identical overall response on the five-category rubric on 67% of the items and were within one 
category on 95% of the items. Further results establishing the reliability and validity of SO h/P are 
provided in the Lewis et al. (1 999) report. 

SCU . A companion instrument to SOM is the Survey of Computer Use (SCU) (Lowther & Ross, 
2001 ). The SCU was completed as part of the SOM observation sessions, where SCU data was also 
recorded in 15-minute intervals and then summarized on an overall data form. 

The SCU was designed to capture exclusively student access to, ability with, and use of 
computers rather than teacher use of technology. Therefore, four primary types of data are recorded: 
(a) computer capacity and currency, (b) configuration, (c) student computer ability and (b) student 
activities while using computers. Computer capacity and currency is defined as the age and type of 
computers available for student use and whether or not Internet access is available. Configuration 
refers to the number of students working at each computer (e.g., alone, in pairs, in small groups). 
Student computer ability is assessed by recording the number of students who are computer literate 
(e.g., easily used software features/menus, saved or printed documents) and the number of students 
who easily use the keyboard to enter text or numerical information. 

The next section of the SCU focuses on student use of computers with regard to: the types of 
activities, subject areas of activities, and software being used. The computer activities are divided into 
three categories based on the type of software tool: production tools, Internet/research tools, and 
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educational software. Within each category, primary types of software are identified. For example, 
under Production Tools, the software includes: word processing, databases, spreadsheets, 
draw/paint/graphics, presentation (e.g., PowerPoint ™), authoring (e.g., KidPix ™), concept mapping 
(e.g., Inspiration), and planning (MS Project ™). For the Internet/research tools, three types of 
software are included: Internet browser, CD reference materials, and communications (e.g., email, 
listservs, chat rooms). The Educational Software also has three types of software: 
drill/practice/tutorial, problem-solving (e.g., Riverdeep ™) and process tools (e.g., Author’s Toolkit ™). 
With this type of recording system, several activities can be noted during the observation of one 
student working on a computer. For example, if a student gathered data from the Internet, created a 
graph from the data, then imported the graph into a PowerPoint presentation, the observer would 
record three types of software tools as being observed: Internet browser, spreadsheet, and 
presentation. This section ends by identifying the subject area of each computer activity. The 
categories include: language arts, mathematics, science, social studies, other, and none. The 
computer activities and software being used are summarized and recorded using a five-point rubric 
that ranges from (0) Not Observed to (4) Extensively observed. 

The final section of the SCU is an “Overall Rubric” designed to assess the degree to which the 
activity reflects “meaningful use” of computers as a tool to enhance learning. The rubric has four 
levels: 1 - Low-level use of computers, 2 - Somewhat meaningful, 3 - Meaningful, and 4 - Very 
meaningful. 

RSCA . The Rubric for Student-Centered Activities was developed by CREP (Lowther, Ross, & 
Plants, 2000) as an extension to SOM and SCU. The RSCA was used by observers to more closely 
evaluate the degree of learner engagement in seven selected areas considered fundamental to the 
goals of increasing student-centered learning activities (cooperative learning, project-based learning, 
higher-level questioning, experiential/hands-on learning, student independent inquiry/research, 
student discussion, and students as producers of knowledge using technology). These strategies 
reflect emphasis on higher-order learning and attainment of deep understanding of content and 
whether or not technology was utilized as a component of the strategy. Such learning outcomes 
seem consistent with those likely to be engendered by well-designed, real-world linked exercises, 
projects, or problems utilizing technology as a learning tool. Each item includes a two-part rating 
scale. The first is a four-point scale, with 1 indicating a very low level of application, and 5 
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representing a high level of application. The second is a Yes/No option to the question: “Was 
technology used?” with space provided to write a brief description of the technology use. The RSCA 
was completed as part of SOM/SCU observation periods. 

Teacher Surveys 

Three surveys were used to collect impressions of the STARK Program: the School Climate 
Inventory (SCI), the Teacher Technology Questionnaire (TTQ), and the Technology Skills 
Assessment (TSA). Each of the seven participating schools administered the surveys at a faculty 
meeting conducted in May, 2002. The surveys are described below. 

SCI . Researchers at the Center for Research in Educational Policy developed the School in 

1 989 (Butler & Alberg, 1 991 ). The main purpose of the instrument is to assess impacts of reform 
initiatives in relation to seven dimensions logically and empirically linked with factors associated with 
effective school organizational climates. The inventory contains 49 items, with 7 items comprising 
each scale. Responses are scored through the use of Likert-type ratings ranging from strong 
disagreement (1) to strong agreement (5). Each scale yields scores ranging from 7 to 35, with higher 
scores being more positive. Additional items solicit basic demographic information on respondents. 

Face validity of the school climate items and logical ordering of the items by scales were 
established during the development of the inventory (Butler & Alberg, 1991). Subsequent analysis of 
responses collected through administration of the inventory in a variety of school sites substantiated 
validity of the items. Scale descriptions and current internal reliability coefficients on the seven scales 
of the inventory, obtained using Cronbach’s alpha are as follows: 



School Climate Inventory Internal Reliability and Scale Descriptions 



Scale 


Internal 

Reliability 


Description 


Order 


a=.8394 


The extent to which the environment is ordered and 
appropriate student behaviors are present 


Leadership 


a=.8345 


The extent to which the administration provides instructional 
leadership 


Environment 


a=.8094 


The extent to which positive learning environments exist 


Involvement 


a=.7582 


The extent to which parents and the community are involved 
in the school 


Instruction 


a=.7453 


The extent to which the instructional program is well 
developed and implemented 


Expectations 


a=.7275 


The extent to which students are expected to learn and be 
responsible 


Collaboration 


a=.7417 


The extent to which the administration, faculty, and students 
cooperate and participate in problem solving 
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TTQ . The Teacher Technology Questionnaire is a three-part instrument used to collect teacher 
perceptions of computers and technology. In the first section, teachers rate their level of agreement 
with 20 statements regarding five technology-related areas: impact on classroom instruction, impact 
on students, teacher readiness to integrate technology, overall support for technology in the school, 
and technical support. Items are rated with a five-point Likert-type scale that ranges from (1 ) Strongly 
Disagree to (5) Strongly Agree. Two primary questions are asked in the second section. The first 
asks teachers to rate their level of computer ability as very good, good, moderate, poor, or no ability. 
Next, teachers indicate if they have a home computer, and if they do, if they use the home computer 
to access instructional materials on the Internet and/or to prepare classroom materials. 

TSA . The Technology Skills Assessment (TSA) is a 57-item survey that includes 50 three-point 

Likert-type questions and 6 multiple-choice questions. The three-point questions are designed to 
assess the perceived technological abilities of the participants, while the multiple choice questions are 
designed to determine the actual knowledge of the participants on several areas of technological 
literacy. All of the questions are arranged into seven categories, which are aligned to the International 
Society for Technology in Education’s (ISTE’s) National Educational Technology Standards (NETS) 
and the Texas Essential Knowledge and Skills (TEKS) Technology Applications for Grades 3-5 (see 
Appendix A). The categories of the survey are as follows: Computer Basics, Software Basics, 
Multimedia Basics, Internet Basics, Advanced Skills, Using Technology for Learning, and Policy and 
Ethics. In the final section, participants are asked to identify the amount of training they have received 
through the Brazos-Sabine Foundation and through West Orange Collaborative integration program. 

DATA COLLECTION 

Table 1 provides the type of measures, instruments, number collected, administration timeline, 
and a brief data collection description for each of the instruments described in the previous section. 

Table 1 

Data Collection Summary 




Classroom 

Observations 


SOM 

SCU 

RSCA 


52 

52 

52 


Spring 2003 


• Prearranged one-hour sessions in which 
teachers demonstrated a prepared lesson 
using technology -notes forms were 
completed every 15 minutes of the lesson. 


Surveys 


SCI 


276 


May 2003 


• Administered during a faculty meeting 




TTQ 


243 




held during May 2003 at each of the 1 0 




TSA 


286 




schools 
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RESULTS 



Classroom Observation Results 

The data for 52 classroom observations were collected with SOMs, SCUs, and RSCAs during 
prearranged sessions in which teachers implemented a lesson using technology. Results from each 
measure are described in the section below. This is followed with survey results. 

School Observation Measure (SOM®) 

As indicated in the description of SOM, the observation procedure primarily focused on 24 
instructional strategies using a five-point rubric (0 = not observed, 1 = rarely, 2 = occasionally, 3 = 
frequently, and 4 = extensively). In an initial analysis, we computed the percentage of times a strategy 
was not observed (rubric category = 0) vs. observed (categories 1-4 combined). These results were 
then compared with the Year 1 baseline data. As shown in Table 2, direct instruction was the most 
frequently observed strategy for both years, however, there was a 1 0% decrease in its use during 
Year 2. The second most frequently observed strategy was teacher as a coach/facilitator, which was 
seen in nearly three-fourths of the observations (71 .2%). This was a shift from baseline data in which 
use of higher-level instructional feedback (59.3%) was the second most prevalent strategy. 

Chi-square analysis revealed a significant difference between Year 1 and Year 2 observations 
for approximately 30% (7 of 24) of the strategies. The most relevant were a 50% increase in the use 
of technology as a tool (1 6.8% vs. 65.4%) and a 40% increase in the use of computers for 
instructional delivery (29.6% vs. 67.3%). Other notable changes include a 30% increase in integration 
of subject areas (13.0% vs. 44.2%), and 20% increase in the use of higher level questioning strategies 
(33.4 vs. 53.8) and project-based learning (27.8% vs. 46.2%). The Year 2 data also revealed a 
significant increase in the use of systematic individual instruction (0.0% vs. 23.1%) and individual 
tutoring (11. 3% vs. 32.7%). 
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Table 2 



School Observation Measure (SOM) Proportion of not observed (0) vs. observed (1-4) strategies 

Year 1 (2001-2002) N = 54 
Year 2 (2002-2003) N = 52 



Direct instruction 


13.0 


19.2 


87.1 


78.8 


Teacher as a coach/facilitator 


40.7 


25.0 


57.5 


71.2 


Computer for instructional delivery*** 


68.5 


30.8 


29.7 


67.3 


Technology as a learning tool*** 


81.5 


32.7 


16.8 


65.4 


Experiential, hands on learning 


57.4 


42.3 


42.6 


57.7 


Higher level instructional feedback 


38.9 


40.4 


59.3 


55.8 


Use of higher-level questioning* 


64.8 


40.4 


33.4 


53.8 


Independent seatwork 


42.6 


48.1 


55.5 


51.9 


Project-based learning* 


70.4 


50.0 


27.8 


46.2 


Integration of subject areas*** 


85.2 


53.8 


13.0 


44.2 


Ability groups 


72.2 


53.8 


27.8 


42.3 


Student discussion 


70.4 


59.6 


29.7 


38.5 


Cooperative/collaborative learning 


72.2 


59.6 


27.9 


36.5 


Work centers 


74.1 


65.4 


26.0 


34.6 


Individual tutoring** 


85.2 


61.5 


11.3 


32.7 


Independent inquiry/research 


88.9 


75.0 


11.2 


25.0 


Systematic individual instruction*** 


100.0 


76.9 


0.0 


23.1 


Multi-age grouping 


81.5 


78.8 


18.5 


21.2 


Sustained writing/composition 


85.2 


78.8 


14.9 


19.2 


Sustained reading 


81.5 


80.8 


16.8 


17.3 


Team teaching 


83.3 


76.9 


16.8 


17.3 


Performance assessment 


81.5 


84.6 


14.9 


13.5 


Parent/community involvement 


90.7 


88.5 


7.5 


7.7 


Student self-assessment 


90.7 


92.3 


7.5 


5.8 



*p<.05, **p<.01, ***p<.001 

Note . Sorted from highest to lowest proportion of 2002-2003 Observed 



Table 3 presents the full, five-category breakdown of the Year 1 and Year 2 SOM results. When 
examining the strategies that were viewed frequently or extensively in at least 30% of the 
observations, Year 1 data only revealed four activities while Year 2 data revealed 10 activities. Of 
those ten, the following four were observed in over 50% of the visits: direct instruction (69.3%), 
teacher acting as a coach/facilitator (57.7%), computer for instructional delivery (53.9%), and 
technology as a learning tool (52.0%). The six remaining strategies were as follows: experiential 
hands-on learning (48.1%), ability groups (42.3%), project-based learning (40.4%), higher-level 
questioning strategies (38.5%), independent seatwork (38.5%), and higher-level instructional 
feedback (36.6%). 
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Table 3 



School Observation Measure (SOM) Data Summary 



Year 1 (2001-2002) N = 54 
Year 2 (2002-2003) N = 52 




Instructional Orientation 



Direct instruction (lecture) 


01-02 


13.0 


16.7 


22.2 


9.3 


38.9 




02-03 


19.2 


3.8 


5.8 


38.5 


30.8 


Team teaching 


01-02 


83.3 


1.9 


1.9 


1.9 


11.1 




02-03 


76.9 


0.0 


1.9 


9.6 


5.8 


Cooperative/collaborative learning 


01-02 


72.2 


1.9 


5.6 


11.1 


9.3 




02-03 


59.6 


1.9 


7.7 


11.5 


15.4 


Individual tutoring (teacher, peer, aide, adult 


01-02 


85.2 


5.6 


1.9 


1.9 


1.9 


volunteer) 


02-03 


61.5 


7.7 


9.6 


11.5 


3.8 


Classroom Organization 
Ability groups 


01-02 


72.2 


0.0 


0.0 


1.9 


25.9 




02-03 


53.8 


0.0 


0.0 


1.9 


40.4 


Multi-age grouping 


01-02 


81.5 


0.0 


0.0 


0.0 


18.5 




02-03 


78.8 


0.0 


0.0 


0.0 


21.2 


Work centers (for individuals or groups) 


01-02 


74.1 


1.9 


0.0 


3.7 


20.4 




02-03 


65.4 


1.9 


3.8 


11.5 


17.3 


Instructional Strategies 

Higher level instructional feedback (written or 


01-02 


38.9 


25.9 


13.0 


13.0 


7.4 


verbal) to enhance student learning 


02-03 


40.4 


7.7 


11.5 


23.1 


13.5 


Integration of subject areas 
(interdisciplinary/thematic units) 


01-02 


85.2 


3.7 


1.9 


3.7 


3.7 




02-03 


53.8 


11.5 


7.7 


13.5 


11.5 


Project-based learning 


01-02 


70.4 


0.0 


3.7 


1.9 


22.2 




02-03 


50.0 


0.0 


5.8 


9.6 


30.8 


Use of higher-level questioning strategies 


01-02 


64.8 


11.1 


9.3 


9.3 


3.7 




02-03 


40.4 


5.8 


9.6 


23.1 


15.4 


Teacher acting as a coach/facilitator 


01-02 


40.7 


9.3 


5.6 


25.9 


16.7 




02-03 


25.0 


5.8 


7.7 


34.6 


23.1 


Parent/community involvement in learning 
activities 


01-02 


90.7 


0.0 


0.0 


1.9 


5.6 




02-03 


88.5 


1.9 


1.9 


0.0 


3.8 


Student Activities 
Independent seatwork (self-paced 


01-02 


42.6 


11.1 


14.8 


11.1 


18.5 


worksheets, individual assignments) 


02-03 


48.1 


1.9 


11.5 


21.2 


17.3 


Experiential, hands-on learning 


01-02 


57.4 


3.7 


3.7 


9.3 


25.9 




02-03 


42.3 


1.9 


7.7 


23.1 


25.0 
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Table 3 - continued 




Student Activities (continued) 



Systematic individual instruction (differential 


01-02 


100.0 


0.0 


0.0 


0.0 


0.0 


assignments geared to individual needs) 


02-03 


76.9 


3.8 


3.8 


7.7 


7.7 


Sustained writing/composition (self-selected 


01-02 


85.2 


1.9 


3.7 


5.6 


3.7 


or teacher-generated topics) 


02-03 


78.8 


0.0 


3.8 


3.8 


11.5 


Sustained reading 


01-02 


81.5 


5.6 


5.6 


5.6 


0.0 




02-03 


80.0 


0.0 


5.8 


3.8 


7.7 


Independent inquiry/research on the part of 


01-02 


88.9 


5.6 


0.0 


1.9 


3.7 


students 


02-03 


75.0 


1.9 


5.8 


9.6 


7.7 


Student discussion 


01-02 


70.4 


5.6 


9.3 


11.1 


3.7 




02-03 


59.6 


5.8 


9.6 


23.1 


0.0 


Technology Use 














Computer for instructional delivery (e.g. 


01-02 


68.5 


3.7 


11.1 


5.6 


9.3 


CAI, drill & practice) 


02-03 


30.8 


1.9 


11.5 


21.2 


32.7 


Technology as a learning tool or resource 


01-02 


81.5 


5.6 


3.7 


1.9 


5.6 


(e.g. Internet research, spreadsheet or 
database creation, multi-media, CD ROM, 
Laser disk) 


02-03 


32.7 


3.8 


9.6 


13.5 


38.5 


Assessment 














Performance assessment strategies 


01-02 


81.5 


0.0 


7.4 


1.9 


5.6 




02-03 


84.6 


0.0 


5.8 


3.8 


3.8 


Student self-assessment (portfolios, 


01-02 


90.7 


1.9 


1.9 


0.0 


3.7 


individual record books) 


02-03 


92.3 


0.0 


3.8 


0.0 


1.9 


Summary Items 








Low 


Moderate 


High 


Academically focused class time 


01-02 






0.0 


29.6 


64.8 




02-03 






3.8 


7.7 


76.9 


Level of student attention/ 


01-02 






1.9 


35.2 


59.3 


interest/engagement 


02-03 






3.8 


15.4 


69.2 



Upon examination of the final section of SOM, the data indicate at least a 10% increase in the 
percent of visits in which there was a high level of academically focused time (Yr. 1 = 64.8%; Yr. 2 = 
76.9%). Similar results were seen in the level of student attention, interest, and/or engagement, which 
was rated as high in almost 70% as compared to 60% of the baseline observations (Yr. 1 = 59.3%; Yr. 
2 = 69.2%). 

Inferential results: Year 1 vs. Year 2. To determine whether significant changes occurred 
between Year 1 and Year 2, t-tests for independent samples were performed on each item. We also 
computed Effect Sizes ( ES ) using Cohen’s d formula (Cohen, 1988) to determine the educational 
importance of differences (an ES having an absolute value greater than .25 is considered to be 



Page 1 6 of 53 




educationally important). As shown in Figure 1 and Table 4, there were significant differences on 
seven items, with all strategies showing higher frequency of observation in Year 2. In addition, the 
Effect Size for the seven items ranged from 1 .24 to 0.42. The two items revealing the greatest 
difference were technology as a learning tool (M= 2.22) and computer for instructional delivery (M = 
2.24) - both target strategies of the STARK Program. Also noteworthy, five of the seven significant 
items are supportive of student-centered learning in a technology-supported environment (technology 
as a learning tool, integration of subject areas, use of higher-level questioning strategies, independent 
inquiry/research on the part of students, and systematic individual instruction). 

Figure 1 



SOM: Significant Differences 




U- 


Technology as 
tool (ES = 1.24) 


Computer for 
delivery (ES = 
.93) 


Systematic indiv 
instr (ES = .71) 


Integration of 
subject areas (ES 
= .65) 


Use of higher- 
level questions 
(ES = .65) 


Individual tutorinc 
(ES = .57) 


Independent 
inquiry (ES = .42) 


□ 2001-2002 


0.42 


0.81 


0 


0.34 


0.74 


0.23 


0.26 


■ 2002-2002 


2.22 


2.24 


0.65 


1.16 


1.65 


0.82 


0.73 



Scale: 0 = Not Observed; 4 = Extensively Observed 
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Table 4 



A Summary of SOM Items Showing Significant Differences Between 2001-2002 and 2002-2003 




Technology as learning tool or 
resource 


0.42 


1.06 


2.22 


1.76 


6.29 


0.000*** 


+1.24 


Computer for instructional delivery 


0.81 


1.37 


2.24 


1.68 


4.72 


0.000*** 


+0.93 


Systematic individual instruction 


0.00 


0.00 


0.65 


1.31 


3.59 


0.001*** 


+0.71 


Integration of subject areas 


0.34 


0.98 


1.16 


1.50 


3.27 


0.002** 


+0.65 


Use of higher-level questioning 
strategies 


0.74 


1.20 


1.65 


1.61 


3.24 


0.002** 


+0.65 


Individual tutoring 


0.23 


0.76 


0.82 


1.26 


2.79 


0.007** 


+0.57 


Independent inquiry/research on the 
part of students 


0.26 


0.87 


0.73 


1.35 


2.12 


0.037* 


+0.42 



*p<.05, **p<.01, ***p<.001 



Summary. The SOM results revealed significant changes between baseline and Year 2 
observations. These changes included using a greater variety of instructional strategies as well as 
using more student-centered practices. The largest differences were seen with regard to classroom 
use of technology. During Year 1 , computer use was seen in only 30% or less of the observations, 
whereas Year 2 data revealed computer use in over 65% of the visits. The data suggest that the 
STARK program is having a positive impact on increasing teacher’s use of student-centered learning 
techniques, integration of technology into instruction, and higher levels of academically focused class 
time and student attention. 

Rubric for Student-Centered Activities (RSCA) 

Results address the percentage of sessions in which each RSCA strategy was observed at least 
once, the quality/depth of observed strategy applications, and the percentage of sessions in which 
technology was used with the observed strategy. Because the RSCA was used in targeted 
observations of lessons that were to include the use of technology to support learning, computer use 
was expected to be viewed at sometime during the lesson. 

Observed vs. not observed. The first question on each RSCA item asked whether or not the 
particular strategy was observed. As can be seen in Table 5, all strategies were observed more 
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frequently during Year 2 as compared to Year 1 . The most frequently observed strategy during the 
baseline was higher-level questioning (53.7%), followed by experiential hands-on learning (38.9%). 
These results were reversed for Year 2 in that experiential hands-on learning was the most frequently 
observed (58.5%) and higher level questioning followed at 56.6%. Independent inquiry was the least 
observed strategy for Year 1 and Year 2, however, its use increased from 1 6.6% during Year 1 to 
35.9% during Year 2. 

Ratings. The next set of analyses examined the rubric rating (1 to 4) for each of the strategies, 
when the given strategy was observed. That is, if the strategy was not seen, the associated rating of 
“0” was excluded from the analysis because it would seriously negatively bias the overall computation 
of quality/effectiveness. A descriptive summary of the Year 1 and Year 2 means are also presented in 
Table 5. As shown, the Year 2 mean scores range between approximately 2.40 and 3.06 suggesting 
moderate to moderately high levels of quality/effectiveness. The experiential, hands-on learning 
activities, or those that involved extensive and effective use of manipulatives or hands-on experiences 
and project-based learning were assessed as being the most meaningful (M= 3.06, M = 3.04, 
respectively). Also noteworthy was a moderate increase for students as producers of knowledge 
mean score (Yr. 1 M= 1 .85; Yr. 2 M= 2.65) and percent of times this strategy was observed (Yr. 1 = 
24.1%; Yr. 2 = 43.4%). This suggests a positive trend toward more meaningful and frequent student 
use of computers as productivity tools. On a less positive note, two items received lower ratings 
during the Year 2 observations: cooperative learning (Yr. 1 M= 2.75; Yr. 2 M= 2.40) and experiential 
hands-on learning (Yr. 1 M= 3.29; Yr. 2 M= 3.06). Although the mean scores are above average and 
the differences were not significant, it may be worthwhile to monitor these areas during the next year 
of implementation. 

Inferential results: Year 1 vs. Year 2. Inferential analyses (t test for independent samples) were 
conducted to compare Year 1 to Year 2 rubric ratings. No significant differences were revealed. 
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Table 5 



Rubric for Student Centered Activities Item Ratings by Percentage Observed and Mean Scores 

Year 1 (2001-2002) N = 52 
Year 2 (2002-2003) N = 54 




Cooperative Learning 


01-02 


29.6 


1.9 


11.1 


9.3 


7.4 


2.75 




02-03 


47.1 


15.1 


11.3 


7.5 


13.2 


2.40 


Project-Based Learning 


01-02 


27.8 


3.7 


5.6 


5.6 


13.0 


3.00 




02-03 


47.2 


1.9 


13.2 


13.2 


18.9 


3.04 


Higher-Level Questioning Strategies 


01-02 


53.7 


18.5 


14.8 


9.3 


7.4 


2.11 




02-03 


56.6 


11.3 


17.0 


13.2 


15.1 


2.57 


Experiential, Hands-On Learning 


01-02 


38.9 


1.9 


9.3 


3.7 


24.1 


3.29 




02-03 


58.5 


3.8 


13.2 


17.0 


24.5 


3.06 


Independent Inquiry / Research 


01-02 


16.7 


7.4 


0.0 


1.9 


7.4 


2.56 




02-03 


35.9 


13.2 


3.8 


3.8 


15.1 


2.58 


Student Discussion 


01-02 


33.3 


18.5 


0.0 


5.6 


9.3 


2.17 




02-03 


45.2 


9.4 


7.5 


20.8 


7.5 


2.58 


Students as Producers of Knowledge 


01-02 


24.1 


16.7 


0.0 


1.9 


5.6 


1.85 




02-03 


43.4 


7.5 


15.1 


5.7 


15.1 


2.65 



* Rating scale: 1 = limited application; 4 = Strong application. 



Technology use. As can be seen in Figure 2, technology was used to support RSCA strategies 
to a greater extent this year as compared to last year. In nearly 50% of the observations, technology 
was used to support students engaged in project-based learning (45.3%) and experiential hands-on 
learning activities. Teachers also used technology to support cooperative learning and higher level 
questioning during 40% of the observations. Technology was used less frequently in conjunction with 
independent inquiry and student discussion as these uses were only observed in 30% of the visits. 
The lower occurrence of technology during student discussion is not surprising since this activity often 
occurs as face-to-face rather than online interactions. 
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Figure 2 



Percent of RSCA Activities that Used Technology 




VJ 


Cooperative 

Learning 


Project-based 

Learning 


Higher-Level 

Questions 


Hands-on 

Learning 


Independent 

inquiry 


Student 

Discussion 


□ Year 1 


3.7 


9.26 


1.85 


3.7 


7.41 


1.85 


■ Year 2 


39.6 


45.3 


39.6 


45.3 


28.3 


28.3 



‘Students as Producers of Knowledge is not included because use of technology is a required component. 



Summary. The results show that the RSCA strategies were observed in approximately one-third 
to one-half of the classroom observations - a slight increase (1 6%) from the baseline results. Also 
promising is the positive trend in the quality of strategy implementation - with all mean scores being 
above average. Although not significant, the mean scores for five of the seven RSCA strategies were 
higher - indicating continued growth toward teacher competency with non-traditional instructional 
strategies. The most striking RSCA result is the increased usage of technology to support student- 
centered activities. 

Survey of Computer Use (SCU) 

As with the SOM and RSCA, data from the SCU were collected during prescheduled lessons in 
which teachers were asked to use technology. A summary of the observation results for Year 1 and 
Year 2 is provided in Table 6. Data for the 2001 -2002 Computer Configuration, Testing Software, and 
Computers/Digital Tools Used by Students are not listed in the table as the reporting format changed 
from Year 1 to Year 2. Relevant data from these categories will be discussed in the narrative. 
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Table 6 



Survey of Computer Use (SCU) Data Summary 



Year 1 (2001-2002) N = 54 
Year 2 (2002-2003) N = 51 




Classrooms most frequently had the following number of computers or digital tools. 



None . 2.0 

One . 17.6 

2-4 . 51.0 

5-10 . 5.9 

1 1 or more . 23.5 

Classroom computers were most frequently 

Up-to-date . 64.7 

Aging, but adequate . 29.4 

Outdated/limited capacity . 2.0 

No computers were observed . 2.0 

Classroom computers were most frequently 

Connected to the Internet . 84.3 

Not connected to the Internet . 1 3.7 

No computers were observed . 2.0 



Student Computer Use 

Classroom Computers or digital tools were most frequently used by: 

Few (less than 1 0%) students ■ 27.5 

Some (about 1 0-50%) students ■ 1 3.7 

Most (about 51-90%) students ■ 2.0 

Nearly all (91 -1 00%) students ■ 35.3 

Students did not use computers ■ 1 7.6 

Students most frequently worked with computers or digital tools: 

Alone ■ 45.1 

In pairs ■ 7.8 

In small groups ■ 15.7 

Students did not use computers ■ 27.5 

Student computer literacy skills were most frequently: 

Poor . 7.8 

Moderate ■ 27.5 

Very good ■ 19.6 

Not observed ■ 41 .2 

Student keyboarding skills were most frequently: 

Poor . 9.8 

Moderate ■ 5.9 

Very good ■ 25.5 

Not observed ■ 54.9 



Note . Results marked as are not listed as they were reported in a different format or not collected in 2001-2002. 




Desktop Computers 


2002-2003 


47.1 


2.0 


5.9 


11.8 


25.5 


Laptop Computers 


2002-2003 


52.9 


7.8 


9.8 


9.8 


13.7 


Personal Data Assistants (PDA) 


2002-2003 


88.2 


0.0 


0.0 


0.0 


2.0 


Graphing Calculator 


2002-2003 


88.2 


0.0 


0.0 


0.0 


2.0 


Information Processor (e.g., Alphaboard) 


2002-2003 


90.2 


0.0 


0.0 


0.0 


0.0 


Digital Accessories (e.g., camera, scanner, 
probes) 


2002-2003 


78.4 


0.0 


5.9 


0.0 


5.9 



*Note: Item percentages may not total 100% because of missing input from some respondents. 
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Table 6 continued 




Production Tools 



Word Processing 


2001-2002 


88.9 


0.0 


5.6 


1.9 


3.7 




2002-2003 


76.5 


3.9 


3.9 


0.0 


11.8 


Database 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


100.0 


0.0 


0.0 


0.0 


0.0 


Spreadsheet 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


90.2 


0.0 


0.0 


2.0 


3.9 


Draw/Paint/Graphics 


2001-2002 


98.1 


1.9 


0.0 


0.0 


0.0 




2002-2003 


80.4 


3.9 


2.0 


2.0 


5.9 


Presentation (e.g., MS PowerPoint)** 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


74.5 


2.0 


0.0 


5.9 


9.8 


Authoring (e.g., HyperStudio) 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


100.0 


0.0 


0.0 


0.0 


0.0 


Concept Mapping (e.g., Inspiration) 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


90.2 


0.0 


0.0 


0.0 


3.9 


Planning (e.g., MS Project) 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


90.2 


0.0 


0.0 


2.0 


0.0 


Internet/Research Tools 














Internet Browser (e.g., Netscape)** 


2001-2002 


92.6 


7.4 


0.0 


0.0 


0.0 




2002-2003 


74.5 


0.0 


2.0 


3.9 


11.8 


CD Reference (encyclopedias, etc.) 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


86.3 


0.0 


0.0 


3.9 


0.0 


Communications 


2001-2002 


100.0 


0.0 


0.0 


0.0 


0.0 




2002-2003 


100.0 


0.0 


0.0 


0.0 


0.0 


Educational Software 














Drill/Practice/Tutorial 


2001-2002 


83.3 


3.7 


1.9 


0.0 


11.1 




2002-2003 


78.4 


0.0 


2.0 


5.9 


9.8 


Problem Solving (e.g., SimCity) 


2001-2002 


98.1 


0.0 


0.0 


0.0 


0.0 




2002-2003 


90.2 


0.0 


0.0 


0.0 


2.0 


Process Tools (Geometer's Sketchpad, etc.) 


2001-2002 


94.4 


0.0 


0.0 


0.0 


3.7 




2002-2003 


90.2 


0.0 


0.0 


2.0 


2.0 


Testing Software 














Individualized/Tracked (e.g., Accelerated 














Reader) 


2001-2002 














2002-2003 


88.2 


0.0 


0.0 


0.0 


0.0 


Generic 


2001-2002 














2002-2003 


86.3 


0.0 


0.0 


0.0 


0.0 



" p <.01 




Production Tools 


2001-2002 


90.7 


5.6 


1.9 


0.0 


0.0 


1.9 




2002-2003 


27.5 


11.8 


37.3 


11.8 


7.8 


13.7 


Internet/Research Tools 


2001-2002 


88.9 


7.4 


1.9 


0.0 


1.9 


0.0 




2002-2003 


52.9 


3.9 


17.6 


5.9 


3.9 


5.9 


Educational Software 


2001-2002 


79.6 


5.6 


1.9 


11.1 


5.6 


1.9 




2002-2003 


52.9 


5.9 


21.6 


9.8 


5.9 


0.0 


Testing Software 


2001-2002 
















2002-2003 


68.6 


3.9 


5.9 


2.0 


2.0 


2.0 



Notes . 1 ) Results marked as are not listed as they were reported in a different format or not collected in 2001 -2002. 
2) Item percentages may not total 100% because of missing data. 
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Computer Configuration. There was a dramatic increase in the number of classroom computers 
between Year 1 and Year 2. Last year, the majority (60%) of the classrooms had only one computer, 
whereas this year, approximately 80% of the classrooms had two or more computers. Of those 
computers, most (65%) were considered to be well equipped and up to date. Since Year 1 
connectivity of classroom computers to the Internet was fairly high (83.3%), there was very little 
change in the Year 2 percent of classroom computers reported as being connected to the Internet 
(84.3%). When looking at the types of technology that were in use, desktop and laptop computers 
were the most prevalent as they were seen in approximately 50% of the visits (52.9%, 47.1% 
respectively). There was a moderate increase in the percent of visits in which students displayed a 
high level of computer literacy skills (Year 1=11 .2%; Year 2 = 1 9.6%) and keyboarding skills (Year 1 
= 14.9%; Year 2 = 25.5%). 

Student Computer Activities by Software used. The Year 2 as compared to Year 1 SCU data 
show a greater variety of software applications being used by the students. During Year 1 , only 6 of 
the 14 (without Testing software) listed software tools were seen in use, whereas all but two 
(database and authoring) were observed to some degree during the Year 2 classroom visits. Brief 
descriptions of the different software uses, related subject area content of the activities, and noted 
significant differences derived from chi-square tests of independence are presented below. 

Production tools used by students. As shown in Table 6, the Year 2 observations show students 
using each type of production tool (except databases and authoring) in approximately 1 0% to 25% of 
the visits. The most frequently observed software use was for presentation and word processing 
applications, which were seen in approximately 25% of the visits. Presentation software was the 
only production tool used significantly more during Year 2 as compared to Year 1 (p < .01). However, 
the extent of use was limited in that it was frequently or extensively seen in less than 1 6% of the 
classroom observations. With regard to the subject area focus, the majority of production tool activities 
were in language arts (37.3%). Fewer production tool activities involved social studies (13.7%), 
mathematics (1 1 .8%), other non-core subject areas (1 1 .8%), and science (7.8%). 

Internet/research tools used by students. There was a significant Year 1 to Year 2 increase in 
the percent of observations in which students were using the Internet (p < .01 ) and a slight increase in 
student use of CD references. The research activities covered all subject areas, however, the majority 
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were focused on language arts (17.6%). The remainder of the subject areas were seen in only 
approximately 4% to 6% of the visits. 

Educational software use by students. The use of educational software increased from Year 1 to 
Year 2, however, the change was non-significant. As with production and Internet/research tools, the 
educational software activities were typically focused on language arts (21 .6%). Mathematics 
software was observed in approximately10% (9.8%) of the visits, while science and other subject area 
software was seen in only 5.9% of the observations. 

Testing software use by students. Testing software was a new addition to the SCU for Year 2 
data collection to capture student use of programs such as Accelerated Reader®, therefore Year 1 
data are not reported. As seen, testing software was not observed during the classroom visits. 

Overall Meaningful Use of Computers. The culminating assessment on the SCU was the 
observer’s evaluation of the meaningfulness of the way in which technology was integrated with 
teaching and learning. To do this, they were asked to indicate how often they observed computer 
activities at each level of the rubric; e.g., how often was very meaningful use of computers observed. 
As can be seen in Figure 3 and Table 7, significant differences between Year 1 and Year 2 
observations were found on all four levels of the rating scale. The direction of the data shows 
progression toward more meaningful applications during Year 2. For example, “very meaningful use” 
was at least occasionally observed in 1 5.7% of the Year 2 visits but in only 3.7% of the Year 1 visits (p 
< .05). Likewise, “meaningful” usage was observed occasionally or more in 41 .2% of the Year 2 visits 
as compared to only 3.8% in Year 1 (p < .001). 
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Figure 3 



SCU: Meaningfulness of Computer Use Extensively Observed 




Table 7 



SCU: Overall Meaningfulness of Computer Activities 

Year 1 (2001-2002) N = 54 
Year 2 (2002-2003) N = 51 




Low level use of computers* 


2001-2002 


74.1 


1.9 


0.0 


0.0 


24.1 




2002-2003 


68.6 


3.9 


3.9 


3.9 


5.9 


Somewhat meaningful use of computers* 


2001-2002 


81.5 


7.4 


3.7 


0.0 


3.7 




2002-2003 


54.9 


7.8 


5.9 


9.8 


9.8 


Meaningful use of computers*** 


2001-2002 


94.4 


0.0 


1.9 


0.0 


1.9 




2002-2003 


45.1 


2.0 


11.8 


9.8 


19.6 


Very meaningful use of computers* 


2001-2002 


94.4 


0.0 


0.0 


0.0 


3.7 




2002-2003 


66.7 


3.9 


3.9 


0.0 


11.8 



* p <.05, ** p <.01, *** p <.001 



Summary. The SCU results suggest that progress was made between year 1 and Year 2 
with regard to technology integration efforts. Specifically, students had more access to computers and 
digital tools; computer skills; and were more frequently using a variety of software tools. There were 
significant increases in student use of presentation software and use of the Internet, however, these 
and other software applications were still only seen in about 25% of the visits. The most striking result 
was the significant increase - indicating that teachers have increased their understanding and to use 
technology as a tool to enhance student learning. 



Page 26 of 53 




SURVEY RESULTS 

Three surveys (SCI, TTQ, and TSA) were administered to the teachers during a faculty meeting 
held at each school in late May 2003. Results for the three instruments were examined for Year 1 vs. 
Year 2 differences. Data from the TTQ and TSA were also compared based on the amount of 
technology training received: no or partial training vs. full training. Results of the three surveys are 
presented below. 

School Climate Inventory (SCI) 

Teacher responses to the SCI were fairly positive across Year 1 and Year 2, although there was 
a slight Year 2 increase in five of the seven overall dimension mean scores and the overall mean (see 
Figure 4 and Table 8). The Year 2 means were also slightly higher than all the national norms as 
compared to 5 of the seven Year 1 means. The following items on the Year 2 SCI received nearly 
unanimous agreement from the teachers: “parents are always treated courteously when they call or 
visit the school” (96.7%), “low achieving students are given opportunity for success in the schools” 
(94.6%), and “teachers use a variety of teaching strategies or models” (94.2%). The items with the 
lowest level of teacher agreement were “students participate in solving the problems of the school” 
(29.3%) and “parents are involved in a home and school support network” (41 .3%) 

Inferential Analysis. To determine if significant differences existed between Year 1 and 2, a 
MANOVA was conducted to compare the two sets of data. Table 9 presents a summary of the results 
that depict a significant multivariate effect, F( 7, 427) = 6.40, p < .000. Follow-up analyses of univariate 
effects revealed significant differences on two of the seven dimensions: Involvement (Year 1 M= 3.78; 
Year 2 M= 3.93) and Instruction (Year 1 M= 3.99; Year 2 M= 4.08). However, upon examination of 
the resulting Effect Sizes of the differences, the implied educational impact is moderately high for 
Involvement (ES = +0.28), and limited with regard to Instruction (ES = +0.21 ). 
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Figure 4 



SCI Overall Mean Scores 




* Sorted from highest to lowest Year 2 Mean Scores 



Summary. The moderate gain in SCI results suggests a positive trend in teacher attitudes about 
their school climate. This was particularly true for Involvement and Instruction, which were both 
significantly higher than the previous year. A slight decrease was seen for Order and Environment; 
however, the resulting means were still higher than the national norms. The data also suggest that 
schools may want to explore ways to increase student and parent-level involvement with educational 
issues. Overall SCI results once again reflect school environments that are generally open and 
supportive of reform initiatives focused toward improved educational opportunities. 



Page 28 of 53 



Table 8 



School Climate Inventory Results 



Year 1 (2001-2002) N = 297 
Year 2 (2002-2003) N = 276 




COLLABORATION Items 

1 . The faculty and staff share a sense of commitment to the school goals. 

6. Students are encouraged to help others with problems. 

16. Teachers are encouraged to communicate concerns, questions, and constructive 
ideas. 

26. Students participate in solving the problems of the school. 

28. Faculty and staff cooperate a great deal in trying to achieve school goals. 

31 . Teachers do not participate enough in decision making.** 

40. Most problems facing this school can be solved by the principal and faculty. 

ENVIRONMENT Items 

7. Faculty and staff feel that they make important contributions in this school. 

9. Varied learning environments are provided to accommodate diverse teaching and 
learning styles. 

10. The school building is neat, bright, clean, and comfortable. 

14. School employees and students show respect for each other's individual differences. 

29. An atmosphere of trust exists among the administration, faculty, staff, students, and 
parents. 

38. Teachers are proud of this school and its students. 

49. People in this school really care about each other. 

EXPECTATIONS Items 

2. Low achieving students are given opportunity for success in this school. 

3. School rules and expectations are clearly defined, stated, and communicated. 

1 7. Students share the responsibility for keeping the school environment attractive and 
clean. 

21 . Students are held responsible for their actions. 

22. Many students in this school are not expected to master basic skills at each grade 
level.** 

27. Many students do not participate in classroom activities because of their sex, race, 
religion, socioeconomic status, or academic ability.** 

43. Teachers have high expectations for all students. 

INSTRUCTION Items 

4. Teachers use a variety of teaching strategies or models. 

15. Teachers sequence learning activities so that students can experience success at 
each step. 

24. Teachers provide opportunities for students to develop higher-order skills. 

33. Curriculum guides ensure that teachers cover similar subject content within each 
grade level. 

35. Teachers use appropriate evaluation methods to determine student achievement. 

41 . Pull-out programs often disrupt and interfere with basic skills instruction.** 

48. Teachers use a wide range of teaching materials and media. 

**ltems are negatively worded, therefore a lower score = a more positive result. 



88.9 


93.1 


74.4 


79.0 


79.1 


85.5 


25.9 


29.3 


84.2 


89.5 


34.0 


27.5 


68.4 


65.6 


83.2 


86.6 


81.5 


83.7 


80.5 


72.1 


69.0 


67.8 


57.6 


62.3 


85.5 


84.1 


73.4 


78.3 



90.6 


94.6 


81.5 


87.0 


58.2 


54.7 


66.0 


63.0 


21.2 


19.9 


14.1 


9.8 


81.1 


86.6 


86.9 


94.2 


85.2 


87.7 


83.5 


90.6 


79.8 


86.2 


87.5 


91.7 


26.9 


23.2 


83.5 


92.0 
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Table 8 continued 




INVOLVEMENT Items 

5. Community businesses are active in this school. 

1 1 . Parents are involved in a home and school support network. 

12. Parents are treated courteously when they call or visit the school. 

18. Parents are invited to serve on school advisory committees. 

1 9. Parent volunteers are used wherever possible. 

32. Information about school activities is communicated to parents on a consistent basis. 
37. Parents are often invited to visit classrooms. 

LEADERSHIP Items 

8. The administration communicates the belief that all students can learn. 

20. The administration encourages teachers to be creative and to try new methods. 

34. The administration provides useful feedback on staff performance. 

36. The administrative staff does not do enough to protect instructional time.** 

42. The principal is an effective instructional leader. 

45. The goals of this school are reviewed and updated regularly. 

47. The principal is highly visible throughout the school. 

ORDER Items 

13. Rules for student behavior are consistently enforced. 

23. Student discipline is administered fairly and appropriately. 

25. Student misbehavior in this school interferes with the teaching.** 

30. Student tardiness and absence from school is a major problem.** 

39. The school is a safe and secure place in which to work. 

44. Teachers, administrators, and parents assume joint responsibility for student 
discipline. 

46. Student behavior is generally positive in this school. 



55.2 


54.0 


40.1 


41.3 


92.6 


96.7 


77.1 


80.4 


72.7 


81.2 


78.5 


89.1 


57.6 


67.4 


89.9 


92.8 


82.5 


87.3 


76.4 


87.0 


17.8 


19.2 


74.1 


85.9 


76.4 


77.2 


74.7 


83.7 


57.6 


65.2 


60.9 


57.2 


54.5 


57.2 


37.4 


38.4 


80.1 


78.6 


52.5 


51.8 


62.6 


60.1 



Items are negatively worded, therefore a lower score = a more positive result. 



SCI Section 2 

Dimensions by Overall Means and National Norms 



Collaboration 


3.71 


3.79 


3.74 


Environment 


3.94 


3.87 


3.79 


Expectations 


3.88 


3.90 


3.85 


Instruction 


3.99 


4.08 


4.02 


Involvement 


3.78 


3.93 


3.77 


Leadership 


3.96 


4.08 


3.92 


Order 


3.35 


3.27 


3.22 


OVERALL 


3.80 


3.85 


3.76 
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Table 9 



SCI: Significant Differences Between Last Year and This Year 



Overall Hotellings T F Hypothesis df Error df Significance 

0.10 6.40 7.00 427.0 0.000*** 




INVOLVEMENT 


3.77 


0.62 


3.93 


0.51 


7.84 


0.005** 


+0.28 


INSTRUCTION 


3.98 


0.49 


4.08 


0.48 


5.21 


0.023* 


+0.21 



Teacher Technology Questionnaire (TTQ) 

The Teacher Technology Questionnaire (TTQ) was designed to capture teacher perceptions 
regarding their personal beliefs and practices regarding five areas: impact of technology on classroom 
instruction, impact of technology on students, teacher readiness to integrate technology, overall 
support for technology, and technical support. A total of 268 teachers completed the survey. 

As seen Table 10, the Year 2 results show a positive trend as compared to Year 1 . Each of the 
five TTQ areas had an increase in the percent of teachers agreeing or strongly agreeing with the 
items. The area showing the greatest percent of change was Teacher Readiness to Integrate 
Technology, which had a 22% increase in the percent of teachers who agreed or strongly agreed with 
the items (Yr. 1 = 49.4%; Yr. 2 = 71 .5%). Specifically, approximately three-fourths of the teachers 
agreed or strongly agreed that their “computer skills were adequate to conduct classes that have 
students using technology” (75.7%), that they had “received adequate training to incorporate 
technology into [their] instruction” (72.8%), and that they “know how to meaningfully integrate 
technology into lessons” (71 .6%). This is a shift from last year’s results that indicated teachers were 
most agreeable concerning the positive impact of technology on students (65.0%). 

The two most highly rated items on the survey were from Technical Support. Nearly all of the 
teachers agreed that they “can readily obtain answers to technology-related questions” (81 .0%), and 
that “most of the school computers [were] kept in good working condition” (79.5%). For both years, 
teachers had the lowest level of agreement with the “impact of technology on classroom Instruction” 
(Yr. 1 = 44.0%; Yr. 2 = 54.9%). 
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Table 10 



Teacher Technology Questionnaire (TTQ) 



Year 1 (2001-2002) N = 243* 
Year 2 (2002-2003) N = 268* 




Impact on Classroom Instruction 


01-02 


02-03 


01-02 


02-03 


01-02 


02-03 


1 4. My teaching is more student-centered when technology is 
integrated into the lessons. 

1 6. 1 routinely integrate the use of technology into my instruction. 

1 8. Technology integration efforts have changed classroom learning 
activities in a very positive way. 

20. My teaching is more interactive when technology is integrated into 
the lessons. 

Overall 


14.0 

30.0 
9.9 

14.8 

17.2 


14.9 

18.7 

6.0 

18.3 

14.5 


49.0 

28.8 

28.8 

44.4 

37.8 


32.5 

29.1 

25.0 

29.1 
28.9 


35.8 

39.5 

60.9 

39.9 

44.0 


51.1 

50.4 

67.2 

50.7 

54.9 


Impact on Students 

3. The use of computers has increased the level of student 


9.9 


10.8 


26.3 


20.1 


63.4 


67.9 


interaction and/or collaboration. 

8. The integration of technology has positively impacted student 


9.1 


7.1 


24.3 


19.4 


65.0 


71.6 


learning and achievement. 

1 0. Most of my students can capably use computers at an age- 


7.0 


7.5 


18.5 


15.7 


73.7 


75.7 


appropriate level. 

1 9. The use of technology has improved the quality of student work. 


11.9 


11.9 


30.0 


32.5 


58.0 


53.7 


Overall 


9.5 


9.3 


24.8 


21.9 


65.0 


67.2 


Teacher Readiness to Integrate Technology 

5. 1 know how to meaningfully integrate technology into lessons. 


17.3 


9.0 


31.7 


17.9 


50.2 


71.6 


9. 1 am able to align technology use with my district's standards- 


14.0 


9.3 


38.3 


22.8 


46.1 


65.7 


based curriculum. 

11. 1 have received adequate training to incorporate technology into 


25.5 


10.1 


27.6 


16.4 


46.1 


72.8 


my instruction. 

1 2. My computer skills are adequate to conduct classes that have 


18.5 


9.7 


25.5 


13.1 


55.1 


75.7 


students using technology. 

Overall 


18.8 


9.5 


30.8 


17.6 


49.4 


71.5 


Overall Support for Technology in the School 

4. Parents and community members support our school's emphasis 


6.2 


6.3 


29.2 


30.2 


64.2 


61.2 


on technology. 

13. Teachers receive adequate administrative support to integrate 


14.0 


6.0 


28.4 


15.7 


57.2 


76.9 


technology into classroom practices. 

1 5. Our school has a well-developed technology plan that guides all 


17.3 


11.9 


33.3 


33.2 


49.0 


52.6 


technology integration efforts. 

1 7. T eachers in this school are generally supportive of technology 


7.4 


6.3 


19.3 


17.9 


72.8 


75.0 


integration efforts. 

Overall 


11.2 


7.6 


27.6 


24.3 


60.8 


66.4 


Technical Support 

1 . Most of our school computers are kept in good working condition. 


9.1 


9.3 


9.9 


10.4 


80.2 


79.5 


2. 1 can readily obtain answers to technology-related questions. 


12.8 


6.3 


14.8 


11.6 


71.6 


81.0 


6. My students have adequate access to up-to-date technology 


19.8 


17.2 


24.7 


17.5 


52.7 


62.3 


resources. 

7. Materials (e.g., software, printer supplies) for classroom use of 


23.0 


22.0 


21.4 


14.9 


55.6 


61.2 


computers are readily available. 

Overall 


16.2 


13.7 


17.7 


13.6 


65.0 


71.0 



*Note: Item percentages may not total 100% because of missing input from some respondents. 
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TTQ Section 2: Participant Information 

Percentages by Categories 

Note: Percentages may not total 100% because of missing input from some respondents. 



How would you rate your level of computer ability? 


Very Good 


16.9 


16.4 




Good 


25.5 


41.4 




Moderate 


45.7 


34.7 




Poor 


11.5 


6.3 




No Ability 


0.0 


0.0 


Do you own a home computer? 


Yes 


89.7 


89.2 




No 


9.1 


10.1 


If yes, do you use your home computer to access 


Yes 


71.6 


72.8 


instructional materials on the Internet? 


No 


26.1 


19.7 


If yes, do you use your home computer to prepare 


Yes 


66.1 


69.5 


instructional materials? 


No 


30.3 


21.8 



Even though two-thirds of the teachers were in agreement (67.2%) with regard to the positive 
impacts of technology on classroom learning, only about half of the teachers agreed that they routinely 
integrate technology into their instruction (50.4%), or that their teaching was more student-centered 
(51 .1%) or Interactive (50.7%) when technology is integrated 

Responses to the last items on the TTQ indicate that teacher familiarity and confidence with 
computers is increasing. When asked to rate their level of computer ability, there was a 1 5% increase 
in the number of teachers who rated their ability as very good to good. However, there were no 
notable differences in the number of teachers who had home computers, used the home computers to 
access instructional material on the web, or used them to prepare classroom materials. 

Inferential results: Year 1 vs. Year 2. Inferential analyses, using MANOVA, confirmed the 
impression of more favorable Year 2 responses. The MANOVA, treating the five survey 
categories (impact on classroom instruction; impact on students; teacher readiness; overall 
support; and technical support) as dependent measures, was highly significant, F( 5, 459) = 

1 0.1 9, p < .0001 . Follow-up univariate analyses yielded significance on four of the five categories 
(see Table 1 1 and Figure 5). The area indicating the greatest degree of significant difference was 
“teacher readiness to integrate technology” (Yr.1 M = 3.40; Yr. 2 M= 3.85), which had an Effect 
Size of .58. Two other areas had Effect Sizes greater than 0.25: Impact on classroom instruction 
(ES = -0.27), and overall support ( ES = -0.27). 
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Figure 5 



TTQ Significant Differences 




Table 11 

TTQ: Significant Differences Between Last Year and This Year 



Overall 


Hotellings T 


F 


Hypothesis df 


Error df 


Significance 




0.11 


10.19 


5.00 


459.00 


0.000*** 




Readiness to Integrate Technology 


3.40 


0.80 


3.85 


0.76 


38.76 


0.000*** 


0.58 


Impact on Classroom Instruction 


3.32 


0.75 


3.53 


0.78 


8.26 


0.004** 


0.27 


Overall Support 


3.60 


0.62 


3.77 


0.63 


8.55 


0.004** 


0.27 


Technical Support 


3.61 


0.76 


3.77 


0.75 


5.56 


0.019* 


0.21 



Inferential results: No or Partial Training vs. Full Training. Inferential analyses, using MANOVA, 
were also conducted to compare teachers who had received no or partial technology training vs. those 
who had completed the technology training and were among those whose classes were observed. 

The MANOVA again treated the five survey categories (impact on classroom instruction; impact on 
students; teacher readiness; overall support; and technical support) as dependent measures. The 
analysis revealed significant differences between the trained and partially trained teachers F(5,184) = 
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3.64, p < .004 (see Table 1 2). The follow-up univariate analyses yielded significance on two of the 
five categories (Figure 6 and Table 12) both with Effect Sizes exceeding .50. The first area was 
“Teacher readiness to integrate technology,” with the fully trained teachers (n = 33) reporting 
significantly more agreement (no/partial M= 3.84; full M = 4.22) than teachers with no or partial 
training (n = 201 ). These teachers also reported significantly more agreement with regard to the 
“Impact of technology on classroom instruction (no/partial M= 3.49; full M= 3.93) than teachers who 
had received no or partial technology training. 

Figure 6 



TTQ: Significant Differences between 
No/Partial and Full Training 




Table 12 

TTQ: Significant Differences Between No or Partial Training and Full Training 

Overall Hotellings T F Hypothesis df Error df Significance 

099 064 OOO 184.00 0.004** 




Teacher Readiness to Integrate 



Technology 


3.84 


0.74 


4.22 


0.75 


5.95 


0.016* 


+0.51 


Impact on Classroom Instruction 


3.49 


0.76 


3.93 


0.93 


7.22 


0.008** 


+0.57 



*p<.05, ***p<.001 
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Summary. The TTQ results for 2002-2003 show significant positive changes from the previous 
year in teacher perceptions of their readiness to integrate technology. This readiness is reflected in 
the high level of reported confidence with regard to teacher computer skills being adequate enough to 
conduct classes that have students use technology. There was also strong agreement that the 
teachers could readily obtain answers to technical problems and that school computers were kept in 
good working condition. However, approximately 50% of teachers were still less positive with regard 
to the impact of technology on their instruction. When examining the TTQ results by amount of 
technology training, two significant differences were revealed - both suggesting that the training had 
an educationally sound impact on teacher readiness to integrate technology and teacher belief that 
technology has a positive impact on classroom instruction. 

Technology Skills Assessment (TSA) 

The primary purpose for the TSA was to assess teacher perceptions of their technology ability as 
noted in the TEKS Technology Applications for grade 3-5 students. The survey begins by asking the 
teachers to rate “How easily. . .” (Not at all, Somewhat, Very Easily) they could use or complete 47 
computer-related tasks divided into six basic areas: computers, software, multimedia, Internet, 
advanced skills, and using technology for learning. Each of these areas had one multiple-choice 
question that required the teachers to demonstrate knowledge by selecting the correct response. The 
teachers were also asked to rate “How well” they understood three technology-related policy and 
ethics (Not at all, Somewhat, Very well) items. A summary of the results from 285 TSA surveys is 
presented in Table 13. 

Teacher confidence was once again the highest for Computer Basics, with slightly over 80% of 
the Year 2 teachers as compared to 16% of the Year 1 teachers indicating that they could Very Easily 
(80.2%) complete each item. The same was seen for the multiple-choice item for this area, with 
91 .5% of the teachers selecting the correct response. A majority of the teachers also reported a high 
level of knowledge with regard to Policy and Ethics (65.1%) and Software Basics (61.1%), however 
less than half (37.9%) of the teachers responded correctly to the related multiple choice item. Only 
about one-fourth of the teachers indicated they could very easily complete the tasks that required 
advanced skills (26.4%) or those that used technology for learning (25.7%) 
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Table 13 



Technology Skills Assessment Data Summary 




Computer Basics ~ How easily can you ... 


01-02 


02-03 


01-02 


02-03 


01-02 


02-03 


I.Use a spell check tool. 


5.6 


2.9 


14.0 


8.1 


80.0 


88.6 


2.Create basic computer documents (word processed) in a timely manner. 


6.0 


3.3 


23.9 


14.0 


69.5 


82.0 


3. Use help menus for software programs. 


6.3 


3.3 


38.6 


21.7 


53.7 


74.3 


4. Use basic computer terms like mouse, keyboard, hard drive, CD-ROM, and 


1.8 


0.0 


15.8 


7.7 


81.4 


91.9 


monitor. 

5. Save documents so they can be opened on both a Macintosh and PC. 


20.7 


11.4 


29.1 


25.4 


49.1 


62.1 


6.Create folders on a hard drive or disk. 


16.1 


7.4 


35.1 


18.8 


47.4 


73.2 


7.Save files to specific folders. 


13.3 


4.8 


33.0 


22.1 


52.6 


73.2 


8.Locate and delete unwanted files. 


7.7 


2.6 


25.6 


19.1 


65.3 


77.2 


9. Use keyboard commands to cut, copy, or delete text. 


11.2 


5.5 


28.8 


20.2 


59.6 


74.3 


lO.Proficiently use a mouse and keyboard. 


1.4 


0.0 


15.8 


8.5 


81.8 


91.5 


1 1 .Print a document using "Print" from the File menu and/or the toolbar icon. 


2.1 


1.1 


9.1 


4.4 


88.1 


93.4 


Computer Basics: Overall 


8.4 


3.8 


24.4 


15.5 


66.2 


80.2 


Software Basics 














1 2.Use software preview features to check work. 


17.2 


7.7 


30.5 


22.8 


51.6 


68.8 


13.0pen and use software programs that are installed on your computer. 


3.5 


1.5 


29.5 


18.4 


66.0 


79.4 


14. Work with and move between two open programs (e.g., Internet and 


30.9 


11.8 


30.9 


31.6 


38.2 


56.3 


database) to create a product. 

15. Describe the difference between downloading and installing software. 


21.4 


10.3 


36.1 


30.5 


42.1 


58.5 


16.Save documents so they can be opened in a different program (e.g., from 


33.7 


14.7 


32.6 


34.6 


33.0 


49.6 


Word to Word Perfect). 
17.lnstall software. 


26.3 


14.7 


30.2 


26.8 


40.0 


53.7 


Software Basics: Overall 


22.2 


10.1 


31.6 


27.5 


45.2 


61.1 


Multimedia Basics 














18.lmport digital video from a camera to a computer. 


62.1 


47.4 


21.1 


26.1 


16.5 


25.7 


1 9.Record and save your voice onto a computer. 


72.6 


57.7 


18.9 


22.8 


8.1 


19.5 


20. Use a scanner to import a photo or document into a computer. 


49.8 


39.7 


25.6 


23.9 


23.9 


35.7 


21 .Play a music CD on the computer. 


11.2 


7.4 


22.1 


13.6 


66.0 


78.7 


Multimedia Basics: Overall 


24.7 


38.1 


26.1 


21.6 


48.4 


39.9 


Internet Basics 














22.Connect to the Internet with a modem (phone, cable). 


18.2 


11.8 


24.2 


11.8 


57.2 


76.1 


23.Use Boolean strategies for Internet searches. 


62.1 


39.0 


20.0 


21.7 


15.8 


37.1 


24. Use appropriate software and the Internet to find audio, video, and 


38.2 


16.5 


36.5 


37.5 


24.6 


44.9 


graphics for lesson plans. 

25. Use the Internet to find help when you have a computer problem. 


43.9 


26.5 


33.7 


35.7 


22.1 


36.4 


26. Determine if information you find on the Internet is accurate and valid. 


40.0 


23.9 


41.1 


39.7 


18.9 


35.7 


27,Evaluate Internet search strategies to determine those that are most 


41.4 


24.3 


38.2 


35.3 


20.4 


39.3 


efficient. 

28.Determine the usefulness and appropriateness of digital information. 


54.7 


33.8 


32.3 


33.1 


12.3 


32.0 


Internet Basics: Overall 


42.6 


25.1 


32.3 


30.7 


24.5 


43.1 
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Table 13 - continued 
Advanced Skills 



29. Use more advanced computer terms like megahertz, gigabytes, and RAM. 


53.0 


38.6 


31.2 


39.3 


14.4 


21.0 


30.Access information on local area networks (LANs) and wide area networks 


60.0 


39.0 


26.3 


34.2 


11.6 


26.1 


(WANs). 

31 .Use appropriate digital layout and design to meet the needs of defined 


68.1 


47.8 


20.7 


32.0 


8.4 


18.8 


audiences. 

32. Use appropriate digital layout and design for the selected media (e.g., 


67.0 


45.6 


22.8 


32.0 


8.4 


21.3 


multimedia, web, print). 

33. Publish information in a variety of media (e.g., printed, monitor display, 


61.1 


38.6 


28.1 


35.3 


8.4 


23.9 


web-based, video). 

34.Connect a computer to a local server to share files. 


67.0 


47.1 


23.9 


26.8 


7.7 


24.6 


35.Determine if a software program works with an operating system. 


64.9 


48.5 


23.5 


26.5 


9.1 


23.2 


36.Print to a specific printer when connected to a network that has more than 


46.0 


28.7 


28.8 


26.1 


23.5 


44.1 


one printer. 

37,Use presentation software to share information with specific audiences. 


58.6 


30.1 


24.6 


32.0 


14.4 


34.9 


Advanced Skills: Overall 


60.6 


40.4 


25.5 


31.6 


11.8 


26.4 


Using Technology for Learning 














38.Use multimedia software to enhance learning experiences. 


42.8 


18.8 


38.6 


42.6 


16.5 


36.4 


39. Use appropriate software (e.g., word processing, graphics, databases, 


38.2 


16.5 


36.8 


43.8 


22.8 


38.2 


spreadsheets, simulations, and multimedia) to express ideas and solve 
problems. 














40. Use text and graphics to create and modify solutions to problems. 


52.6 


26.1 


28.8 


41.9 


16.8 


30.9 


41 .Use digital audio and video to create and modify solutions to problems. 


70.2 


44.1 


21.8 


37.9 


6.3 


16.5 


42.Use communication tools to participate in group projects. 


60.0 


33.5 


24.9 


39.0 


11.9 


25.4 


43. Manipulate information in interactive digital environments (e.g., simulations, 


71.6 


50.4 


20.0 


30.5 


6.0 


17.6 


virtual labs, field trips). 

44. Participate in a listserv, chat, and bulletin board session. 


62.1 


41.2 


23.9 


25.4 


11.9 


32.0 


45.Create an electronic teaching portfolio to evaluate your work. 


74.7 


52.2 


17.9 


29.8 


5.3 


16.2 


46.Evaluate electronic portfolio products. 


74.7 


52.2 


18.9 


30.9 


3.5 


14.7 


47.Create technology tools to assess student work (e.g., checklists, timelines, 


58.9 


31.3 


26.3 


37.9 


11.6 


28.7 


rubrics). 














Using Technology for Learning: Overall 


60.6 


36.6 


25.8 


36.0 


11.3 


25.7 


Policy and Ethics 














48.My school's acceptable use policy. 


6.0 


1.8 


25.3 


18.0 


64.6 


76.8 


49.The concept of a school site license for software. 


9.5 


5.9 


30.2 


24.3 


56.5 


66.2 


50. How to determine if it is legal to copy a software program or another 


18.6 


9.9 


38.2 


34.2 


38.6 


52.2 


individual's electronic work. 














Policy and Ethics: Overall 


11.4 


5.9 


31.2 


25.5 


53.2 


65.1 



Section 2 Multiple Choice Questions 



Responses to multiple-choice questions 

Correct answers to the multiple-choice questions are in BOLD. 




Under which menu item is the "New" How should a Macintosh-created word processed document be 

document option typically found? named so it can be opened on a PC? 





01-02 




02-03 






01-02 


02-03 


File 


256 


89.8% 


249 


91.5% 


Document, wp 


30 


10.5% 


25 


9.2% 


Edit 


3 


1.1% 


4 


1 .5% 


Document.mac2pc 


84 


29.5% 


85 


31 .3% 


View 


3 


1.1% 


6 


2.2% 


Document.doc 


74 


26.0% 


103 


37.9% 


Insert 


1 


0.4% 


1 


0.4% 


Document.exchng 


21 


7.4% 


13 


4.8% 
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Table 13 - continued 




Which of the following is a format used to Which of the following is a well-known Internet search browser? 

save graphics? 







01-02 




02-03 






01-02 


02-03 


grf 


82 


28.8% 


56 


20.6% 


Microsoft.com 


22 


7.7% 


12 


4.4% 


drw 


6 


2.1% 


4 


1 .5% 


Google.com 


172 


60.4% 


234 


86.0% 


gif 


96 


33.7% 


140 


51.5% 


lnfo.com 


13 


4.6% 


2 


0.7% 


img 


42 


14.7% 


37 


13.6% 


Searchlnternet.com 


37 


13.0% 


8 


2.9% 




If you were asked to describe the size of a You have decided to have your students determine which of three 

hard drive , which measure would be most cities has the greatest rainfall during the month of February , and 

appropriate to use in your description? provide the results in the form of a graph. Which of the following 



would be the best type of software for the students to use. 





01-02 




02-03 






01-02 


02-03 


quadrite 


2 0.7% 


1 


0.4% 


Database 


73 


25.6% 


55 


20.2% 


qiqabvtes 


183 64.2% 


200 


73.5% 


Spreadsheet 


139 


48.8% 


176 


64.7% 


Megahertz 


58 20.4% 


44 


16.2% 


Organizer 


9 


3.2% 


11 


4.0% 


Pixels 


1 0.4% 


0 


0.0% 


Publisher 


22 


7.7% 


5 


1 .8% 



Technology Training 



How many Foundation modules have you completed? 




01-02 


02-03 


None 


63.2% 


33.1% 


Less than 5 


24.2% 


3.3% 


5-10 


1.1% 


12.9% 


More than 1 0 


0.4% 


40.8% 


How many West Orange Technology Training sessions have you completed? 




01-02 


02-03 


None 


37.5% 


36.0% 


One 


13.3% 


7.7% 


Two 


14.7% 


15.4% 


Three 


1 1 .2% 


6.3% 


Four 


6.3% 


4.4% 


Five 


0.7% 


0.7% 


More than Five 


8.1% 


16.2% 



Technology Training. There was over a 50% Year 1 (01 .5%) to Year 2 (53.7%) increase in the 
number of teachers indicating that they had completed 5 or more Foundation Program Modules. 
However, there was only a 2% increase in the percent of teachers reporting that they had completed 
two or more West Orange Technology training sessions. 

Inferential results: Year 1 vs. Year 2. A MANOVA comparing the means of Year 1 and Year 2 on 
the seven TSA categories (computer, software, multimedia, Internet, advanced skills, using 
technology for learning, and policy and ethics) yielded a highly significant difference, F( 7, 409) = 7.47, 
p = .0001 . Univariate analysis of variance (ANOVA) was then separately performed on each 
category. As seen in Figure 7 and Table 14, significant differences with Effect Sizes ranging from .33 
to .64 were found across the seven areas. The greatest degree of significant difference was for the 
category “using technology for learning” (Yr.1 M= 1 .52; Yr. 2 M= 1 .91), which had an Effect Size of 
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.64. A similarly high Effect Size (ES = 59) was found for Internet Basics (Yr.1 M= 1 .86; Yr. 2 M = 
2.23). 

Figure 7 



TSA: Significant Differences between 2001-2002 and 2002-2003 




Table 14 

TSA: Significant Differences Between Last Year and This Year 



Overall 


Hotellings T 


F 


Hypothesis df 


Error df 


Significance 




0.13 


7.47 


7.00 


409.00 


0.000*** 




Computer Basics 


2.62 


0.41 


2.76 


0.37 


13.23 


0.000*** 


0.36 


Software Basics 


2.28 


0.60 


2.53 


0.54 


20.31 


0.000*** 


0.44 


Multimedia Basics 


1.81 


0.56 


2.02 


0.63 


13.07 


0.000*** 


0.35 


Internet Basics 


1.86 


0.62 


2.23 


0.64 


36.17 


0.000*** 


0.59 


Advanced Skills 


1.54 


0.61 


1.88 


0.69 


28.52 


0.000*** 


0.52 


Using Technology 


1.52 


0.56 


1.91 


0.66 


42.92 


0.000*** 


0.64 


Policy and Ethics 


2.44 


0.58 


2.62 


0.50 


10.65 


0.001*** 


0.33 



Inferential results: No or Partial Training vs. Full Training. As with the TTQ, A MANOVA, was 
once again conducted to compare teachers who had received no or partial technology training vs. 
those who had completed the technology training. The MANOVA treated the seven TSA survey 
categories as dependent measures. The analysis revealed significant differences between the trained 
and partially trained teachers F(7,1 76) = 2.51 , p < .01 8 (see Table 1 5). The follow-up univariate 
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analyses yielded significance on five of the seven categories (Figure 8 and Table 15). The area 
showing the most significant difference (ES = .72) was once again “using technology for learning 
(no/partial M = 1 .83; full M= 2.30). The remainder of the differences all had Effect Sizes greater than 
.40, thus indicating a strong educational impact. 

Figure 8 



TSA: Significant Differences between 
No/Partial Training and Full Training 




Table 15 

TSA: Significant Differences Between No/Partial Training and Full Training 



Overall 


Hotellings T 


F 


Hypothesis df 


Error df 


Significance 




0.99 


2.51 


7.00 


176.00 


0.018* 




Computer Basics 


2.74 


0.38 


2.89 


0.17 


4.23 


0.041* 


0.42 


Multimedia Basics 


1.96 


0.63 


2.25 


0.64 


4.89 


0.028* 


0.46 


Internet Basics 


2.18 


0.66 


2.47 


0.53 


4.63 


0.033* 


0.45 


Advanced Skills 


1.82 


0.69 


2.14 


0.68 


4.82 


0.029* 


0.46 


Using Technology 


1.83 


0.65 


2.30 


0.64 


12.13 


0.001*** 


0.72 



Summary. The TSA results revealed an overall increase in teacher confidence to complete a 
wide variety of computer-related tasks. The greatest level of confidence was with computer, software, 
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Internet and multimedia basics. The teachers who had completed the technology training were 
significantly more confident with their ability to use technology as a learning tool. 



CONCLUSIONS 

The conclusions of the study will be presented in association with each of the major research 
questions in the respective sections below. 

To what degree and how is technology integrated with classroom instruction by teachers in 
general* and those who have received Foundation and/or integration training through the 
Enhanced Learning Academy? 

Classroom observation data from both the SOM and SCU revealed significant differences in the 
degree and the way in which teachers integrate technology into their teaching. The most striking was 
a 40% increase in the use of computers as an instructional delivery tool and the 50% increase in 
student use of technology as a learning tool, which was seen in over 65% of the classroom 
observations. Another noteworthy increase in the use of computers is seen in data from the RSCA, 
which revealed that teachers integrated technology with approximately one third or more of their 
student-centered activities, as compared to less than 1 0% the previous year. Although students had 
more access to computers and digital tools, significant increases in the types of computer activities 
observed were only found for the use of Internet and presentation software. Word processing and 
educational software were the other two most frequently observed applications. Despite the limited 
scope of technology tools that were used to support classroom instruction, a dramatic improvement 
was seen in teacher ability to create and implement activities that meaningfully integrated the use of 
technology to enhance student learning. Specifically, meaningful (from somewhat to very meaningful) 
use of computers was only seen in 9.3% of the Year 1 observations as compared to 41 .2% in Year 2. 
These results are highly suggestive that the STARK Program is effectively impacting technology 
integration efforts in the target schools. 

* Note . The Year 2 classroom observations were once again conducted during pre-arranged 
visits to classrooms of teachers who had received technology training, therefore, the results reflect 
trained teachers rather than teachers in general. The Year 3 data will include all teachers. 
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To what degree do integration-trained teachers use methodologies that, consistent with 
STARK Program goals, stress higher-order learning and student-centered learning activities? 

In over two-thirds of the classroom observations, teachers assumed the role of coaches or 
facilitators and technology was being used as a tool to support learning. Other activities that foster 
critical thinking and student engagement were seen in over 50% of the visits. These included 
experiential hands-on learning (57.7%), teacher provision of higher-level instructional feedback to 
students (55.8%), and teacher use of higher-level questioning strategies (53.8%). Although no 
significant increases were found when specifically examining the quality with which teachers 
implement student-centered activities, it is promising to note that all mean scores were above average 
- with five being higher than Year 1 ratings. The data indicate continued growth toward teacher 
competency with non-traditional instructional strategies. However, professional development efforts 
may want to increase the focus on better preparing teachers to implement these methods to increase 
the frequency with which they are utilized. 

To what degree have teachers acquired the technology skills specified in the Texas State 
standards? 

Impressive increases were seen in teacher confidence in all seven categories of technology 
skills. Nearly all (90% or more) of the teachers indicated that they could easily complete basic 
computer and software tasks and that they understood the policy and ethics of technology use. 
Three-fourths (75%) percent felt confident when using the Internet and over 60% reported that they 
could easily or very easily complete multimedia and/or advanced tasks, and use technology for 
learning. As would be expected, the teachers who had completed the technology training were 
significantly more confident with their ability to use technology as a learning tool - again an indication 
of successful teacher training through the STARK Program. 

What are school outcomes in school climate and teacher uses of and attitudes toward 
technology? To what extent do these variables (a) reflect Foundation training and STARK 
Program goals, and (b) correlate with one another and with implementation success? 

Results from the School Climate Inventory were again above national norms, suggesting that the 
environment of 1 0 the participating schools was supportive of school-wide initiatives such as 
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technology integration. This positive climate was evidenced in the significant increases in frequency 
of classroom use of technology and teacher ability to create and implement meaningful computer 
activities. Positive teacher attitudes are also reflected in the TTQ results, which showed significantly 
higher teacher readiness to integrate technology and more confidence to conduct classes that have 
students use technology. A contributing factor to this increased confidence could perhaps be the 
increased level of computer literacy as reported on the TSA, plus the ready access to technical 
support provided by the schools and district. Significant differences were again revealed when 
examining results by the degree of training received - suggesting that the STARK Program had an 
educationally sound impact on teacher readiness to integrate technology and teacher belief that 
technology has a positive impact on classroom instruction. The overall results do suggest a positive 
trend in achieving the STARK Program goals and successful implementation, however the current 
efforts need to continue or increased to impact the approximately 50% of teachers who were still less 
positive with regard to the impact of technology on their instruction. 

What factors appear most instrumental in determining schools’ success at achieving the goals 
and overall implementation of the STARK Program? 

Below is a list of key factors from the Year 2 study that appeared to have influenced the progress 
being made toward achievement of the STARK Program implementation and goals. As seen, the 
factors cross key elements that are critical for program success: 

Significant differences in instructional practices 

- Increased use of student-centered learning 
Increased use of computers for instructional delivery 
Increased use of technology as a learning tool 

Significant differences in the quality of instructional practices 

- Increased use of very meaningful computer activities 
Significant differences in school climate 

Increased level of parent and community involvement 

- Increased level of instruction (e.g., more variety, media resources, higher-order learning) 
Significant differences in teacher attitudes and beliefs regarding technology integration 

- More teachers felt ready to integrate technology 
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- More teachers believed technology positively impacts classroom instruction 

- More teachers agreed technology efforts are well-supported 
Significant differences in teacher computer skills 

- More teachers reported confidence with technology tasks 

Overall, the triangulation of positive changes suggests a strong synergy between the diverse 
factors that resulted in impressive progress toward achieving the STARK Program goals during the 
second year of implementation. As seen, the schools had positive climates, teacher technological 
competence increased, and classroom practices engaged students in meaningful technology- 
supported activities. These positive results suggest that program implementation for Year 3 should 
emulate Year 2, but with increased intensity to ensure all teachers are impacted. 
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